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Abstract 

Contemporary  military  campaigns  increasingly  count  on  the  use  of  air  power. 
Suppression  of  enemy  air  defenses  (SEAD)  operations  have  been  a  crucial  element  of 
military  air  power  for  50  years.  Several  developments  and  evolution  in  both  air  defense 
and  attack  systems  suggest  that  SEAD  missions  will  continue  to  have  growing 
importance  to  air  forces.  Since  SEAD  operations  have  a  significant  impact  on  air 
campaigns,  it  is  important  to  examine  their  efficiency  and  identify  improvement 
opportunities.  This  study  explores  factors  that  influence  SEAD  operations  through  use  of  a 
discrete  event  simulation  built  in  Arena  and  subsequent  statistical  analysis  of  the  results. 
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DISCRETE  EVENT  SIMULATION  OF  A  SUPPRESSION  OF  ENEMY  AIR 


DEFENSES  (SEAD)  MISSION 


I.  Introduction 


1.1  Background 

Since  the  first  use  of  aircraft  in  combat,  the  ways  to  defend  forces  on  the  ground 
has  been  a  great  challenge  to  the  armed  forces.  There  are  reports  of  balloon  and  anti¬ 
balloon  artillery  in  the  American  Civil  War  and  the  Franco-Prussian  War,  and  in  1890  the 
Russians  tested  a  field-gun  battery  against  a  balloon  moored  three  kilometers  away.  The 
first  airplane  downed  in  combat  fell  to  ground  fire  in  the  Italo-Turkish  War  of  1912;  so 
when  World  War  I  began,  there  were  precedents  for  ground-based  air  defense  (Werrell, 
1988:  1). 

Small  anns  and  artilleries  were  used  to  hit  the  aircraft  during  World  War  I.  On  the 
other  side  to  make  air  defenses  inoperative,  aircraft  could  have  made  only  strafing  and 
bombing  operations.  Since  that  time,  the  activities  of  neutralizing,  destroying,  or 
temporarily  degrading  enemy  air  defenses  has  been  known  as  suppression  of  enemy  air 
defenses  (SEAD)  which  led  to  the  design  and  construction  of  aircraft  systems  and 
weapons  for  that  purpose.  Over  the  years,  both  attacking  aircraft  and  air  defense  systems 
have  evolved.  German  forces  densely  used  anti-aircraft  artilleries  (AAA)  during  WWII. 
The  Allies  tried  several  ways  to  neutralize  the  German  AAA,  but  the  most  effective 
solution  was  avoidance.  With  the  advent  of  radio  detecting  and  ranging  (radar) 
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equipment,  ground-based  air  defenses  became  more  effective  and  more  lethal.  Towards 
the  end  of  WWII,  Germany  attempted  to  develop  a  surface-to-air  missile  (SAM),  but  the 
technology  necessary  to  provide  guidance  for  a  SAM  was  not  mature  enough  (Neufeld, 
1995:  152).  Therefore,  AAA  continued  to  be  the  primary  threat  and  avoiding  AAA 
continued  to  be  the  primary  tactic  throughout  the  Korean  War.  Especially  in  the  Vietnam 
War,  the  Soviet-built  radar  guided  SA-2  SAM  added  a  significant  lethal  dimension  to  air 
defense.  Total  combat  losses  due  to  ground-based  air  defense  systems  and  the  growing 
rate  of  attrition  provided  clear  evidence  that  SEAD  missions  were  highly  important  for 
maintaining  aircraft  survivability  and  led  to  an  increase  in  the  number  of  planned  SEAD 
sorties.  This  resulted  in  the  development  of  new  SEAD  missions  and  tactics  against  the 
evolving  threat.  Crucial  steps  in  the  evolution  of  the  SEAD  mission  to  actively  jam 
enemy  air  defense  systems  included  introduction  of  the  EB-66  electronic  warfare  (EW) 
aircraft  and  employment  of  the  first  Wild  Weasel  SEAD  aircraft,  the  F-100F  carrying  the 
AGM-45A  Shrike  anti -radiation  missile  (ARM). 

Afterwards  SEAD  missions  took  an  important  role  in  Arab-Israeli  Wars.  In 
contrast  to  Vietnam’s  single  threat,  Israelis  fought  against  an  air  defense  umbrella 
consisting  of  a  variety  of  systems,  many  with  the  ability  to  minimize  the  effects  of 
electronic  counter  measures  (ECM)  such  as  jamming.  In  1982,  two  important  steps  in  the 
evolution  of  air  defense  and  SEAD  mission  were  demonstrated  during  the  Bekaa  Valley 
conflict  between  Israel  and  Syria.  The  Syrians  constructed  a  complicated  integrated  air 
defense  system  (IADS).  SAM  and  AAA  sites  were  placed  to  build  a  forceful  defense  wall 
against  attacks.  In  response,  the  Israelis  developed  a  new  tactic  in  the  SEAD  mission. 
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They  used  a  combination  of  drones  and  aircrafts.  Drones  were  flown  as  decoys  to  make 
SAM  radars  active  and  then  SEAD  aircrafts  were  used  to  employ  standoff  weapons. 

The  next  major  application  of  the  SEAD  mission  was  the  Gulf  War.  As  opposed 
to  Syrian  air  defenses  in  1982,  Iraq  had  gathered  an  impressive  amount  of  sophisticated 
equipment  for  their  IADS,  including  both  Soviet  and  European  systems.  It  consisted  of 
several  thousand  radars,  approximately  10,000  pieces  of  AAA,  up  to  17,000  SAMs,  and 
the  seventh  largest  air  force  in  the  world  (Brungess,  1994:  38).  The  major  concern  was  to 
destroy  or  disrupt  command  and  control  centers,  communication  and  electrical  facilities 
of  the  Iraqi  IADS  instead  of  directly  attacking  the  SAM  sites.  The  SEAD  packages  were 
formed  of  F-4G  high-speed  anti-radiation  missile  (HARM)  shooters,  EA-6B  electronic 
jammers,  and  a  large  number  of  drones  to  support  other  air  strikes.  The  air  campaign 
resulted  in  a  disintegrated  Iraqi  IADS  in  the  first  two  days  by  destroying  or  making 
inoperative  many  of  the  radars  and  SAM  sites. 

The  last  major  example  of  SEAD  operations  was  one  of  the  most  challenging  of 
SEAD  missions.  In  Kosovo,  Serbians  perfonned  new  tactics  that  they  learned  from 
Iraqi’s  experience.  Instead  of  continuously  operating  their  systems,  they  chose  to  change 
the  locations  of  their  mobile  SAMs  continuously  and  activate  them  intennittently.  That 
fact  protected  their  SAMs  from  exposure  to  NATO  attacks.  It  also  gave  Serbian  SAMs 
the  chance  to  launch  surprise  attacks  on  the  Allied  Forces  aircraft,  resulting  in  the  loss  of 
an  F-l  17  and  F-16.  This  was  resulted  in  that  although  strike  aircraft  were  not  always 
threatened,  there  was  a  requirement  for  a  full  complement  of  NATO  SEAD  assets 
airborne  to  support  every  strike  package  (Lum,  1999:  38). 
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Since  the  first  use  of  aircraft  in  combat  and  the  first  response  given  from  the 
ground,  it  was  obvious  that  the  fight  between  aircraft  and  air  defense  would  continue  for 
a  long  time.  This  is  evident  today  with  the  continuing  development  of  new  weapons  and 
improved  tactics.  Going  forward  into  the  21st  century  SEAD  missions  will  continue  to 
mature  with  specialized  aircraft  to  execute  these  important  parts  of  the  air  campaign. 

1.2  Research  Problem 

Contemporary  military  campaigns  increasingly  count  on  the  use  of  air  power. 
SEAD  operations  have  been  a  crucial  element  of  military  air  campaigns  for  50  years. 
Several  developments  and  evolution  in  both  air  defense  and  attack  systems  suggest  that 
SEAD  missions  will  continue  to  have  growing  importance  to  air  forces.  Twenty  to  thirty 
percent  of  all  combat  sorties  in  the  recent  three  major  conflicts  were  devoted  to  SEAD 
missions  (Bolkcom,  2005:  5).  Since  SEAD  operations  have  a  significant  impact  on  air 
campaigns,  it  is  a  necessity  to  detennine  their  efficiency  and  improvement  opportunities. 

1.3  Research  Objective 

This  study  describes  a  method  for  modeling  SEAD  air  combat  operations  in  a 
discrete  event  simulation  environment.  The  objective  of  this  research  is  to  present  a 
flexible  and  responsive  model  by  using  discrete-event  simulation  to  investigate  the  means 
of  neutralizing,  degrading,  jamming  or  destroying  ground-based  air  defense  systems. 
Researching  the  efficiency  of  missions  and  commenting  on  the  results  for  different 
scenarios  are  additional  objectives  of  this  study. 
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1.4  Thesis  Organization 

This  thesis  is  organized  in  five  chapters.  Chapter  two  reviews  simulation 
literature,  combat  modeling,  and  previous  studies  on  related  subjects.  Chapter  three 
defines  the  structure  of  the  model,  how  it  is  built  in  Arena®,  and  gives  some  detailed 
information  of  the  model.  In  chapter  four,  model  results  and  conclusions  are  presented. 
The  last  chapter  pulls  together  highlights  from  all  chapters  and  makes  some  conclusions 
and  recommendations  for  future  research. 
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II.  Literature  Review 


2.1  Systems  and  Models 

A  system  is  defined  to  be  a  collection  of  entities  or  components  that  interact  with 
each  other  and  with  the  environment  in  an  attempt  to  achieve  some  goal  (Hartman,  1985). 
Military  systems  fall  into  this  defined  category.  The  entities  or  components  of  the 
military  systems  might  be  aircraft,  weapons,  troops,  or  various  sized  units  such  as 
squadrons  or  battalions. 

Systems  can  be  categorized  in  two  types,  discrete  and  continuous.  A  discrete 
system  is  one  for  which  the  state  variables  change  instantaneously  at  separated  points  in 
time.  A  continuous  system  is  one  for  which  the  state  variables  change  continuously  with 
respect  to  time  (Law,  2007:  70).  If  an  aircraft  is  taken  into  consideration,  it  moves 
through  the  air  in  continuous  time,  but  it  can  be  modeled  using  a  discrete  event  model  to 
gain  the  convenience  of  computer  programming  and  efficiency  of  computer  operation 
(Hartman,  1985).  Only  a  few  systems  are  totally  discrete  or  continuous  but  can  typically 
be  modeled  as  either  to  achieve  the  objectives  of  the  study. 

We  often  usually  study  complex  systems  to  discover  the  characteristics  of  how 
they  operate.  A  common  objective  in  these  studies  is  to  analyze  the  behavior  of  the 
systems  when  different  conditions  or  inputs  are  applied.  With  these  studies  we  can  gain 
information  about  the  internal  processes  and  relationships  between  the  components  of  the 
systems.  Thus  we  can  make  some  predictions  about  the  performance  of  the  systems  under 
new  and  untested  conditions.  Figure  1  (Law,  2007:  4)  shows  different  ways  in  which  a 
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Figure  1.  Ways  to  study  a  system  (Law,  2007:  4) 
system  might  be  studied.  Our  discussion  focuses  on  cases  where  we  cannot  experiment 
with  the  actual  system.  If  it  is  possible  and  cost-effective  to  build  a  physical  model  of  the 
system  this  can  be  the  best  way  to  get  valid  results  for  system  performance  under  new 
conditions.  For  many  systems  such  as  military  operations,  it  is  not  feasible  to  build  a 
physical  model  of  the  system  being  studied.  For  these  reasons,  the  behavior  of  military 
systems  by  means  of  mathematical  modeling  is  studied. 

A  model  of  a  real  system  is  a  representation  of  some  of  the  components  of  the 
system  and  of  some  of  their  actions  and  interrelationships  which  is  useful  for  describing 
or  predicting  the  behavior  of  the  system  (within  a  reasonable  range  of  inputs)  (Hartman, 
1985).  When  using  a  model,  an  important  question  to  be  answered  is  the  validity  of  the 
model.  Since  no  model  can  represent  the  real  system  perfectly,  how  closely  it  reflects  the 
system  and  the  accuracy  of  the  outputs  in  regards  to  the  model’s  purpose  are  the  main 
issues  for  validity.  Validity  will  be  discussed  in  more  detail  later. 
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After  deciding  to  construct  a  mathematical  model  of  a  system,  the  next  step  is  to 
determine  whether  an  analytical  solution  or  simulation  is  more  appropriate.  If  the  model 
is  simple  enough,  exact  analytical  solutions  can  be  reached.  But  if  an  analytical  solution 
to  a  mathematical  model  is  not  available  or  if  such  a  solution  requires  a  large  amount  of 
time  and/or  other  resources,  simulation  emerges  as  the  preferred  method.  Since  most 
military  systems  are  highly  complex,  it  is  generally  impossible  to  model  them  using  an 
analytical  approach.  Therefore,  simulations  are  used  in  the  analysis  of  military  systems. 

2.2  Combat  Models  and  Their  Classification 

As  defined  before,  a  model  is  a  simplified  representation  of  some  components  of  a 
system  and  some  of  their  interactions  which  is  useful  in  describing  or  predicting  the 
behavior  of  the  system.  A  combat  model,  usually  a  simulation  model,  is  specialized  to 
capture  elements  of  military  operations  for  investigative  purposes  or  resources 
management  purposes  (Miller:  Class  handouts,  OPER  671).  It  is  useful  and  helpful  to 
classify  combat  models  for  a  better  understanding.  Although  there  are  several  ways  to 
make  this  classification,  Hartman’s  (1985)  classification  is  used  to  classify  them  in  this 
study. 

Dynamic  vs.  Static 

A  static  model  represents  a  system  at  only  a  particular  time,  or  represents  a 
system  where  time  has  no  effect.  On  the  other  hand,  a  dynamic  model  represents  a  system 
where  time  clearly  plays  a  role.  Monte  Carlo  models  and  a  model  of  the  lethality  of  a 


single  missile  could  be  given  as  examples  of  static  models.  Most  operational  models  and 
modeling  of  air  combat  are  dynamic  models. 

Continuous  vs.  Discrete 

In  continuous  models,  state  variables  change  continuously  with  respect  to  time.  In 
discrete  models,  the  state  variables  change  instantaneously  at  separate  points  in  time.  In 
other  words,  the  system  can  change  at  only  a  countable  number  of  points  in  time.  A 
discrete  model  can  be  used  to  model  a  continuous  system.  Many  combat  processes  are 
continuous,  but  can  be  modeled  using  a  discrete  event  model.  The  specific  objectives  of 
the  study  and  the  preference  of  the  personnel  programming  the  simulation  are  the  main 
reasons  in  selecting  a  discrete  model  over  a  continuous  model  or  vice  versa.  Although 
there  are  several  examples  of  simulation  software  such  as  Simulink®  and  ACSL  for 
building  continuous  models,  the  discrete-event  simulation  (DES)  package  Arena®  as 
well  as  other  commercial  DES  packages  have  continuous  modeling  capabilities  as  well. 

Deterministic  vs.  Stochastic 

If  a  model  does  not  contain  any  probabilistic  components  or  random  effects,  it  is 
called  detenninistic.  In  a  stochastic  model,  there  is  always  some  random  input  or  process. 
If  a  missile  is  shot  with  the  same  parameters  each  time  and  it  reaches  the  target  in  the 
same  way,  this  model  is  detenninistic.  If  the  impact  point  is  not  known,  then  the  accuracy 
of  the  missile  might  be  modeled  stochastically.  A  model  can  have  both  detenninistic  and 
stochastic  inputs  in  different  components  to  simulate  both  the  certainty  and  randomness 
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of  real  life.  If  any  portion  of  a  system  is  modeled  stochastically,  the  output  of  the  model 
is  also  stochastic. 

Descriptive  vs.  Prescriptive 

A  descriptive  model  describes  how  a  system  will  operate  if  values  for  all  of  the 
input  variables  and  decision  rules  are  given  by  the  model  user  (Hartman,  1985). 

Queueing  models,  inventory  models  and  most  combat  simulation  models  are  descriptive. 
A  weaponeering  program  used  to  evaluate  different  munitions  against  a  specific  target  to 
achieve  the  highest  probability  of  kill  (Pk)  is  an  example  of  a  descriptive  model.  A 
prescriptive  model  specifies  how  the  system  ought  to  operate  to  achieve  some  objective 
(Hartman,  1985).  Prescriptive  models  are  optimization  problems  with  decision  variables 
determined  by  solving  the  model  for  the  given  parameters  of  the  problem.  Linear 
programming,  integer  programming  and  network  problem  models  are  prescriptive 
models.  A  weaponeering  program  could  also  be  used  as  a  prescriptive  model  if  you  allow 
the  model  to  select  a  weapon/target  pairing  given  an  objective  function  and  constraints. 

High  Resolution  vs.  Aggregated 

Combat  models  can  also  be  classified  by  scope.  Combat  models  are  typically 
grouped  using  a  multi-tiered  or  hierarchical  family  of  models.  This  model  hierarchy 
(Figure  2)  is  often  displayed  as  a  pyramid  (Miller:  Class  handouts,  OPER  671). 

In  this  model  hierarchy,  combat  models  are  placed  at  levels  based  on  resolution 
and  aggregation.  Resolution  is  the  degree  of  detail  and  precision  used  in  the 
representation  of  real  world  aspects  in  a  model  or  simulation  (Department  of  Defense, 
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1995).  A  high  resolution  combat  model  (engineering  level)  is  a  combination  of  detailed 
interactions  of  individual  combatants  or  weapon  systems.  The  lowest  level  of  the  pyramid 
contains  engineering  levels  of  the  detailed  system  representations.  The  next  level  presents 
the  system  as  a  combination  of  these  detailed  sub-units  and  includes  the  details  of  an 
engagement  between  a  small  number  of  platforms. 


Figure  2.  Combat  Model  Hierarchy 

Above  that,  the  mission  level  contains  models  where  the  systems  begin  to  interact 
with  a  larger  number  of  other  systems.  This  level  represents  multiple-unit  engagements  or 
battles.  These  kinds  of  models  give  the  operational  performance  of  the  systems.  At  the 
top  of  the  pyramid,  we  find  aggregated  or  low  resolution  combat  models  developed  to 
model  combat  at  the  campaign  level.  An  aggregated  combat  model  is  a  model  of  larger 
units  gathered  from  individual  combatants  with  the  loss  of  some  detailed  information.  At 
this  level,  a  major  theatre  war  including  joint  and  coalition  forces  over  an  extended 
period  of  time  could  be  modeled. 

At  the  bottom  of  the  pyramid,  high  resolution  models  show  the  detailed 
representation  of  combat  and  represent  small  units.  As  we  move  up  the  pyramid,  some 
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details  are  left  out  and  the  models  become  more  abstract  and  entities  begin  to  represent 
larger  units.  Similarly,  as  we  move  from  bottom  to  the  top  of  the  pyramid  the  stochastic 
structure  of  high  resolution  models  shifts  into  a  more  deterministic  type  of  aggregated 
models. 

Weapon  versus  passive  target  models  are  generally  engineering  level  models  to 
discover  the  accuracy  and  lethality  of  a  weapon  system  against  particular  targets  by 
emphasizing  its  hardware  characteristics.  One-on-one  or  few-on-few  models  are  usually 
stochastic  models  between  representing  weapon  systems  in  simplified  engagement 
scenarios  to  represent  the  tradeoffs  between  the  weapon  systems.  Combined  arms  task 
force  models  are  generally  stochastic  and  high  resolution  models  that  represent  individual 
combatants  and  their  detailed  interactions  at  battalion  level.  The  emphasis  of  these 
models  is  to  detennine  the  contribution  of  a  particular  system  to  overall  force 
effectiveness.  Mission  specialty  models  represent  a  high  resolution  of  a  particular  aspect 
or  capability  of  a  unit  while  considering  the  remaining  capabilities  of  the  same  unit  in 
less  detail.  Division  level  force  models  emphasize  the  force  structure  and  the  command 
and  control  functions  of  a  division,  since  a  division  is  the  lowest  level  organization  which 
has  its  own  fire  support  and  logistics.  Campaign  models  have  the  largest  number  of 
participants  including  land,  air,  and  naval  combatants.  The  scenarios  can  last  for  months, 
thus  deployment  and  logistics  sides  of  war  should  be  taken  into  consideration  in  these 
models.  These  models  are  highly  aggregated  and  often  deterministic  with  an  emphasis  on 
logistics,  allocation  and  command  and  control  of  forces. 
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2.3  The  Uses  and  Purposes  of  Combat  Models 

Combat  models  have  a  wide  range  of  variety  and  uses.  Today  many  countries’ 
armed  forces  use  combat  modeling.  Combat  modeling  as  a  tool  for  decision  making  can 
provide  a  more  economic  and  effective  means  to  evaluate  alternatives  and  as  an  aid  in 
determining  appropriate  force  structures  and  capabilities.  In  addition,  combat  models  can 
be  used  to  educate  staff  officers  and  as  a  training  aid  in  many  different  areas. 

As  the  technology  develops,  new  military  weapon  systems  continue  to  improve 
and  the  cost  of  them  continues  to  increase.  While  optimizing  the  design  and  maintaining 
the  quality,  reducing  these  costs  is  a  principle  area  of  concern  for  many  countries. 

Combat  modeling  and  simulation  is  one  of  the  ways  to  approach  this  problem.  New 
weapon  systems  and  justifications  are  usually  modeled  by  high  resolution  models.  These 
models  give  a  high  level  of  description  of  the  new  systems  using  a  variety  of  stochastic 
components.  At  the  same  time,  they  help  in  understanding  the  contribution  of  the  system 
to  mission  effectiveness.  These  models  may  also  be  used  to  help  evaluate  new  and 
modified  tactics  for  the  operators  (Hartman,  1985).  Different  tactical  developments  can 
be  tried  and  evaluated  to  find  out  the  best  or  most  effective  under  various  conditions. 

Combat  models  are  also  used  to  analyze  the  ability  of  different  types  of  forces  and 
major  weapon  systems  for  total  force  structuring.  To  understand  the  contribution  of  an 
existing  unit  or  new  weapon  system,  it  can  be  modeled  in  a  campaign  level  model.  Such 
models  can  be  used  to  evaluate  unit  size  and  composition  to  provide  decision  makers  with 
a  better  idea  about  the  structure  and  capability  of  the  total  force. 
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Another  major  area  where  combat  models  can  be  used  is  for  training  personnel. 
These  models  can  contain  military  tasks  to  be  evaluated  and  practiced  by  a  specific  staff. 
Models  used  for  training  often  run  in  real  time  and  allow  for  human  interaction. 

Distributed  Interactive  Simulation  (DIS)  is  a  simulation  architecture  used  by  the 
military  for  conducting  real-time  platform  level  war  gaming  across  multiple  host 
computers.  It  was  first  designed  in  support  of  the  US  Army  Simulator  Network 
(SIMNET)  program  for  tank  training  by  the  sponsorship  of  the  United  States  Defense 
Advanced  Research  Project  Agency  (DARPA)  in  the  early  1990’s.  DIS  consists  of 
autonomous  simulation  entities  such  as  battlefield,  environment,  and  simulation  support 
entities  interacting  in  real  time  across  networks.  DIS  transmits  only  the  information  for 
change  in  the  state  of  entities  across  networks.  It  provides  an  open  architecture  where 
anyone  can  play.  It  is  operable  among  different,  virtual,  live  and  constructive  simulations. 
It  facilitates  development,  training,  mission  planning  and  rehearsal. 

As  described  above,  different  uses  of  combat  models  intend  to  achieve  particular 
purposes.  Military  analysts  frequently  use  models  to  evaluate  future  combat  systems.  For 
any  combat  modeling  study  there  is  always  a  tradeoff  between  time,  cost  and  risk.  Time 
may  be  the  most  limited  resource  in  searching  for  the  best  answer  for  an  ongoing  combat 
operation  or  for  training  personnel  in  a  specific  task  within  a  constrained  environment. 
Cost-efficiency  is  a  crucial  concern  in  many  Operations  Research  (OR)  studies.  Since 
military  technology  is  the  most  expensive  industrial  area  in  the  world,  achieving  the  best 
capability  at  the  lowest  cost  is  a  great  challenge  for  the  researchers  and  developers.  The 
risk  of  being  unsuccessful  in  combat  clearly  has  a  large  impact  on  combat  attrition. 
Combat  models  can  be  used  to  better  understand  what  factors  affect  the  level  of  risk  and 
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how  to  reduce  it.  Although  a  model  is  not  a  perfect  and  exact  representation  of  the  real 
world,  it  can  still  provide  insight  on  the  relative  merits  of  various  courses  of  action  for  the 
decision  maker. 

Thus,  the  purposes  of  combat  modeling  can  be  summarized  in  two  basic 
categories;  analysis  and  training.  Studies  regarding  development  and  effectiveness  of 
weapon  systems,  force  capability,  and  development  of  tactics,  doctrine,  strategy  and 
policy  are  all  common  analysis  areas.  Another  analysis  area  is  operations  support  tools 
for  helping  to  make  decisions.  In  the  training  or  education  part,  there  are  two  main  parts. 
One  of  them  is  the  skills  development  for  individuals  or  teams  and  the  other  is  exercise 
drivers. 

2.4  Model  Verification,  Validation  and  Accreditation 

One  of  the  most  difficult  concerns  in  modeling  that  developers  or  users  of  these 
models  have  to  face  is  determining  whether  a  model  is  an  accurate  representation  of  the 
actual  system.  This  problem  can  be  solved  by  the  steps  of  verification,  validation  and 
accreditation  (VV&A). 

Model  verification  is  the  process  of  detennining  that  a  model  implementation  and 
its  associated  data  accurately  represent  the  developer's  conceptual  description  and 
specifications  (Department  of  Defense,  1995:  A-8).  Debugging  the  simulation  computer 
program  is  a  simple  form  of  verification.  In  essence,  verification  seeks  to  ensure  that  the 
model  is  built  right. 

Model  validation  is  the  process  of  detennining  the  degree  to  which  a  model  and 
its  associated  data  provide  an  accurate  representation  of  the  real  world  from  the 
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perspective  of  the  intended  uses  of  the  model  (Department  of  Defense,  1995:  A-8).  In 
short,  validation  ensures  that  the  right  model  was  built.  If  a  simulation  is  valid,  then  it  can 
be  used  to  make  decisions  about  the  system.  An  important  point  about  model  validity  is 
that  a  valid  model  for  one  purpose  may  not  be  valid  for  another.  Simulation  models 
should  always  be  built  for  specific  purposes.  Another  important  point  about  validation  is 
that  it  is  not  a  one  time  process  undertaken  at  the  end  of  model  development,  but  an 
ongoing  process  conducted  throughout  model  development. 

Accreditation  is  a  concept  introduced  by  U.S.  Department  of  Defense  (DoD)  in 
recent  years.  It  is  the  official  certification  by  a  model  user  that  a  model,  simulation,  or 
federation  of  models  and  simulations  and  its  associated  data  is  acceptable  for  use  for  a 
specific  purpose  (Department  of  Defense,  1995:  A-8).  Accreditation  assures  that  the 
model  user  takes  responsibility  for  the  decision  to  employ  a  model  for  a  particular 
application  and  to  make  official  conclusions  based  upon  model  results. 

Credibility  is  also  a  related  principle.  If  decision  makers  accept  a  simulation 
model  and  its  results  as  correct  and  are  willing  to  use  the  model  results,  that  model  can  be 
deemed  as  credible.  A  credible  model  is  not  necessarily  valid,  and  it  might  not  be  used  as 
an  aid  in  making  decisions.  In  essence,  credibility  implies  that  the  model  provides 
believable  results  and  is  strongly  influenced  by  model  use  by  other  organizations. 

There  are  four  basic  approaches  for  deciding  whether  a  simulation  model  is  valid. 
Each  of  the  approaches  requires  the  model  development  team  to  conduct  verification  and 
validation  (V&V)  as  part  of  the  model  development  process  (Sargent,  2005).  In  the  first 
approach,  which  is  frequently  used,  the  model  development  team  makes  the  V&V 
determination.  Another  approach  leaves  the  V&V  detennination  with  the  users  of  the 
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model.  The  third  approach  uses  an  outside  team  independent  of  both  developers  and  users 
of  the  model  to  make  the  V&V  decision.  The  last  and  rarely  used  approach  incorporates  a 
scoring  model  with  subjective  scores  or  weights  for  various  aspects  of  the  model  and  then 
accepts  the  model  as  valid  if  overall  score  meets  or  exceeds  some  passing  score. 

Some  of  the  verification  and  validation  techniques  are  presented  here.  Common 
verification  techniques  include  writing  and  debugging  a  simulation  program  in 
subprograms,  reviewing  the  program  with  more  than  one  person,  and  running  the  model 
under  several  sets  of  input  parameters  and  checking  the  results  for  reasonableness.  One  of 
the  most  powerful  techniques  that  can  be  used  to  debug  a  discrete-event  simulation 
program  is  a  trace  (Law,  2007:  249).  In  a  trace,  the  states  of  the  system  are  compared 
with  hand  calculations  to  check  the  operations  of  the  program  continue  as  intended. 

Operational  validation  is  determining  whether  the  simulation  model’s  output 
behavior  has  the  accuracy  required  for  the  model’s  intended  purpose  over  the  domain  of 
the  model’s  intended  applicability  (Sargent,  2005).  There  are  three  basic  approaches  to 
make  these  comparisons.  The  first  one  is  subjective  using  graphical  comparisons  such  as 
histograms,  box  plots  and  scatter  plots.  Confidence  intervals  (Cl)  and  hypothesis  tests  are 
the  remaining  two  approaches  that  provide  more  reliable  and  objective  results.  Both 
confidence  intervals  and  hypothesis  tests  can  be  used  to  compare  means,  variances,  and 
distributions  of  the  model  outputs  against  the  system  outputs. 

2.5  Previous  Research 

Many  simulations  involving  air  combat  are  modeled  using  special  combat 
modeling  software  tools.  These  combat  modeling  tools  are  often  produced  for  only  US 
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release  which  means  limited  application.  The  software  used  in  this  study,  Arena®  is  a 
discrete-event  simulation  (DES)  package  and  has  no  restrictions  for  use  by  Non-US 
students.  Our  discussion  focuses  on  some  research  about  air  defense  and  SEAD  that  does 
not  use  special  purpose  combat  models. 

Measuring  the  effectiveness  of  radar  and  infrared  sensors  in  anti-air  warfare  area 
defense  (Kulac,  1999)  is  an  example  of  component-based  DES  developed  in  Java®  using 
the  Simkit  simulation  package.  Analysis  of  ship  self  air  defense  system  selection  (Turan, 
1999)  is  another  Java®  application  using  the  Modkit  simulation  package. 

A  simulation  analysis  of  a  SEAD  operation  (Haugen,  1998)  is  another  application 
of  Simkit.  Haugen  conducted  a  study  to  evaluate  the  impact  of  intelligence  delay  on  a 
SEAD  operation.  The  results  showed  that  the  effectiveness  of  a  SEAD  operation  is 
sensitive  to  information  delay  but  the  effective  variable  is  the  number  of  allocated  SEAD 
aircraft. 

Unmanned  aerial  vehicles  (UAV)  mission  level  simulation  (Walston,  1999)  is  a 
DES  study  written  in  Java®  using  the  Silk®  simulation  package.  In  that  research,  an 
object  oriented  simulation  was  developed  to  model  the  surveillance  and  active  SEAD 
missions  of  UAVs.  Analysis  examined  the  effect  of  speed,  endurance,  and  weather 
susceptibility  on  UAV  operational  effectiveness  and  the  effects  of  radar  cross  section, 
threat  density,  and  threat  lethality  on  UAV  SEAD  mission  performance. 

Simulation  analysis  of  UAV  (Heath,  1999)  is  another  DES  example  for  an  air 
platform.  Analyzing  mine  avoidance  tactics  for  autonomous  underwater  vehicles  (Allen, 
2004),  dynamic  allocation  of  weapons  and  sensors  to  ground  targets  (Havens,  2002),  and 
waterfront  force  protection  (Childs,  2002)  are  some  other  studies  relating  movement  and 
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detection  in  DES.  Simulation  of  autonomic  logistics  system  sortie  generation  (Faas, 
2003)  and  a  DES  model  for  reusable  military  launch  vehicle  prelaunch  operations 
(Stiegehneier,  2006)  are  some  combat  models  built  with  Arena®  . 
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III.  Methodology 


3.1  Introduction 

This  chapter  describes  the  discrete  event  simulation  model  of  a  SEAD  mission 
built  for  this  research  effort.  It  gives  an  overview  of  a  simplified  scenario  which  SEAD 
missions  are  tasked  to  attack  an  air  defense  system.  The  following  sections  contain  model 
selection,  model  structure  and  description,  and  several  assumptions  made  in  the  model. 

3.2  Model  Selection 

The  purpose  of  building  a  simulation  model  is  to  create  a  tool  that  produces 
necessary  data  for  the  researchers.  Thus,  selecting  a  model  should  be  as  simple  as 
possible,  but  at  the  same  time  it  should  give  a  sufficient  level  of  detail.  The  researcher  has 
two  options  in  this  sense;  one  of  them  is  to  use  an  existing  model  and  the  other  is  to 
develop  a  new  one.  As  mentioned  in  the  previous  chapter,  there  are  not  many  examples 
where  a  researcher  builds  a  combat  model  from  scratch  using  discrete  event  simulation 
software.  On  the  other  hand,  the  Air  Force  Standard  Analysis  Toolkit  (AFSAT)  contains 
a  number  of  legacy  models  designed  to  model  combat  at  the  engagement  and  mission 
level.  One  of  these  models  is  Extended  Air  Defense  Simulation  (EADSIM).  This  model 
is  a  mission  level  simulation  used  to  assess  effectiveness  of  many  defense  systems.  It  can 
be  used  to  model  a  variety  of  scenarios  including  SEAD  missions  and  other  air  defense 
operations.  Another  mission  level  model  is  System  Effectiveness  Analysis  Simulation 
(SEAS)  which  helps  to  assess  the  impact  of  proposed  systems  in  terms  of  high  level 
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combat  outcomes.  However,  these  models  have  important  disadvantages  such  as  being 
very  large  and  complex.  Many  of  these  combat  modeling  tools  were  produced  with 
limited  release  outside  of  the  US  and  is  not  necessarily  available  to  international  students. 

The  simulation  model  in  this  research  was  developed  in  the  Arena®  software 
package  which  is  a  commercial  tool  and  available  to  all  students.  It  is  a  discrete  event 
simulation  model  designed  for  analyzing  the  perfonnance  of  and  the  impact  of  changes 
on  complex  systems  associated  with  supply  chain,  manufacturing,  logistics,  distribution 
and  warehousing,  and  other  areas.  In  the  following  sections,  we  present  how  a  combat 
model  was  built  in  Arena®  and  the  other  details  about  a  SEAD  mission. 

3.3  Model  Description  and  Structure 

Mission  success  is  generally  evaluated  by  two  important  measures  in  air  to  ground 
(A/G)  employment.  These  two  factors  are  target  destruction  and  force  survival.  There  are 
also  several  basic  factors  to  be  taken  into  consideration  while  planning  A/G  missions, 
such  as  enemy  defenses,  terrain,  weather,  target  vulnerability,  force  requirements, 
navigation,  and  formations.  There  is  no  single  approved  solution  to  any  tactical  situation. 
Choosing  reasonable,  unpredictable  tactics  is  the  key  in  planning  any  A/G  mission. 

A/G  missions  can  be  created  by  flight  packages  with  more  than  one  flight  or  type 
of  aircraft.  Each  flight  must  understand  the  mission  objectives  to  be  successful.  There  are 
two  basic  objectives  for  A/G  missions.  These  objectives  are  target  destruction  and  force 
survival  where  they  influence  flight  planning  through  all  phases  of  the  mission.  Factors 
considered  during  the  mission  planning  process  include  mission  objectives  given  in  the 
air  tasking  order  (ATO),  rules  of  engagement  (ROE)  or  special  instructions  (SPINS), 
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intelligence  information,  weather,  terrain,  weaponeering,  navigation,  communication, 
force  requirements,  and  suppression  of  enemy  air  defenses  (SEAD).  If  attackers  are 
tasked  to  enter  a  threat  ring  or  attack  a  threat  site,  they  have  to  plan  the  mission  with 
available  SEAD  assets.  If  there  are  no  SEAD  assets  available  to  be  tasked,  an  alternative 
way  is  tasking  some  of  the  allocated  forces  to  the  SEAD  role.  At  this  point,  the  vital  role 
of  SEAD  missions  and  attacking  a  threat  with  or  without  SEAD  assets  can  be  noticed 
easily. 

Offensive  counter  air  (OCA)  operations  are  aimed  against  essential  targets  of  the 
enemy's  air  power.  These  targets  include  air  defense  control  facilities;  defensive  missile 
complexes;  command,  control,  communications,  and  computer  (C4)  facilities;  airfield 
and  supporting  facilities;  aircraft  on  the  ground;  and  munitions  and  missile  storage  sites. 
OCA  missions  against  air  defense  elements  are  called  suppression  of  enemy  air  defenses 
(SEAD)  which  seeks  to  neutralize,  destroy,  or  temporarily  degrade  enemy  surface  based 
air  defenses  by  disruptive  or  destructive  means.  Disruptive  SEAD  involves  a  temporary 
disruption  of  enemy  air  defense  assets.  Employing  a  high  speed  anti-radiation  missile 
(HARM),  electronic  warfare  (EW),  and  information  attack  (IA)  are  the  execution  types  of 
disruptive  SEAD.  EW  involves  the  use  of  electromagnetic  and  directed  energy  to  control 
the  electromagnetic  spectrum  or  to  attack  the  enemy  such  as  jamming  or  deception,  and 
employment  of  anti-radiation  weapons  or  weapons  using  electromagnetic  energy. 
Destruction  of  enemy  air  defenses  (DEAD)  is  one  step  beyond  suppression  and  includes 
the  physical  destruction  of  enemy  air  defense  assets  through  the  use  of  conventional 
bombs  and  contemporary  weapons  such  as  cruise  missiles.  However,  DEAD  was  not 
explicitly  considered  in  this  study. 
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Building  a  model  is  an  art  and  requires  a  conscious  effort.  The  modeler  must 
make  good  decisions  in  selecting  the  right  functional  relationships,  the  best  modeling 
techniques,  the  right  scenarios,  and  the  sources  of  inputs  to  get  accurate  results  to  aid  the 
decision  maker  in  forming  conclusions  about  the  system  being  modeled.  Thus,  the 
modeler  should  have  knowledge  of  the  simulation  tool  and  an  experience  in  the  military 
operation  to  be  modeled.  After  combining  these  factors,  the  modeler  first  takes  steps  to 
design  the  structure  of  a  combat  model.  These  include  determining  the  purpose  of  the 
study,  generating  the  appropriate  combat  scenario,  defining  the  entities,  their  attributes 
(characteristics)  and  the  events  related  to  them.  Once  the  model  structure  is  defined,  the 
modeler  moves  on  to  execution  details  such  as  battle  initialization,  specific  processes  to 
model  (such  as  search,  movement,  and  detection),  battle  termination,  and  required  model 
outputs. 

This  structure  gives  an  idea  about  the  main  processes  of  a  combat  model  and  how 
these  processes  are  flowing  in  an  existing  model.  A  successful  combat  model  scenario 
usually  creates  entities  which  perfonn  the  main  processes:  movement,  searching, 
detecting  and  engaging.  This  provides  the  same  logic  and  flow  chart  for  each  combatant 
side  of  the  model  (Figure  3). 

A  simplified  SEAD  mission  was  developed  in  Arena®  for  this  research.  This 
model  is  used  to  discover  relationships  and  derive  conclusions  depending  on  input 
parameters.  It  is  designed  for  analysis,  with  no  objective  concerning  training.  The  model 
can’t  be  interrupted  or  given  different  directions  after  execution  begins.  The  current 
version  of  the  model  doesn’t  have  a  Graphical  User  Interface  (GUI).  As  a  result,  all  input 
parameters  must  be  set  directly  in  the  code. 
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Start  Battle 


End  Battle 


Figure  3.  The  Decision  Making  Flow  Chart  (Miller:  Class  handouts,  OPER  671) 

This  model  is  dynamic  and  has  an  event  stepped  time  mechanism.  It  represents 
both  stochastic  and  detenninistic  features  with  its  characteristics.  While  constructing  the 
model,  most  of  the  effort  was  consumed  to  get  a  realistic  as  well  as  a  flexible  model.  But 
building  a  more  realistic  model  means  the  modeler  must  include  more  details.  Thus,  some 
assumptions  were  made  to  keep  the  model  simple  and  responsive.  These  assumptions  will 
be  explained  in  the  following  section. 

3.4  Model  Assumptions  and  Details 

When  executing  an  air  strike  or  an  air-to-surface  offensive  counter  air  mission 
against  specific  targets  in  the  battle  area,  the  mission  commander  needs  different  types  of 
aircraft  and  flights  to  compose  a  traditional  package  to  ensure  minimum  attrition.  SEAD 
and  EW  aircraft,  air  to  air  (A/ A)  and  air  to  ground  (A/G)  flights  are  some  typical 
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examples  of  a  package.  This  model  deals  with  only  the  units  that  carry  and  launch  air  to 
surface  weapons  for  attackers  and  air  defense  units  that  carry  and  launch  surface  to  air 
weapons  for  defenders. 

A  two-sided  (Blue  and  Red)  combat  model  was  built  for  this  research.  The  entities 
created  for  both  Blue  and  Red  are  complete  weapon  systems  that  have  some  attributes 
and  can  move  and  interact  with  entities  from  the  other  side.  The  battle  area  is  defined  to 
be  100x100x5  miles  and  is  represented  in  a  x-y-z  coordinate  system.  The  geographic 
positions  of  both  sides  do  not  have  an  impact  on  the  results  of  the  battle  with  the  Blue 
side  located  on  the  east  side  of  the  area  and  attack  in  the  direction  from  east  to  west.  Also 
there  are  no  obstacles  assumed  to  create  any  terrain  factor.  The  battle  time  is  detennined 
as  10  days.  There  are  three  sorties  flown  each  day  and  after  each  sortie  both  units  are 
regenerated  disregarding  previous  sortie  attrition. 

The  main  model  consists  of  two  major  parts  and  an  additional  part  to  capture  the 
outputs  (Figure  4).  In  the  first  part,  the  Red  SAM  sites,  the  Red  targets  and  the  Blue  A/G 
flights  associated  with  these  targets  are  created.  HARMs,  the  search  and  attack  patterns 
of  both  the  Red  air  defense  units  and  HARMs  and  the  movement  and  attack  phases  of 
Blue  A/G  flights  are  built  in  the  second  part.  The  following  paragraphs  will  go  into  detail 
about  these  two  major  parts. 

In  the  first  part,  the  entities  are  the  units  of  air  defense  systems  for  the  Red  side. 
Basically,  an  air  defense  system  has  different  categories  of  units.  These  include  early 
warning,  air  surveillance,  ground  control  intercept  (GCI),  SAM  system  acquisition,  SAM 
system  fire  control,  AAA  fire  control  radars,  engagement  control  stations,  missile  launch 
stations,  and  long,  medium  or  short  range  SAMs  according  to  the  capability  of  the 
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Figure  4.  Main  model 

system.  Since  HARMs  are  anti-radiation  missiles,  the  radars  are  the  primary  concerns  and 
targets  of  SEAD  flights.  There  are  no  particularly  named  air  defense  systems,  weapons  or 
aircrafts  in  this  model.  All  of  the  players  were  intuitively  created  and  given  their 
important  specifications  only  in  numbers. 

The  Red  air  defense  system  is  tasked  to  defend  an  area  100  x  100  miles  with  two 
SAM  sites.  This  does  not  mean  all  the  area  should  be  covered  by  the  defense  umbrella. 

All  air  defense  units  are  considered  mobile,  but  they  need  to  be  stationary  to  operate.  For 
each  sortie,  the  defense  systems  are  settled  on  a  random  location  to  defend  two,  four,  or 
six  strategic  targets  against  SEAD  and  A/G  flights  (Figure  5).  Those  specific  locations  of 
the  air  defense  systems  were  used  as  the  main  target  positions  for  HARMs  and  missile 
launcher  positions  for  A/G  flights  to  avoid.  The  ranges  of  SAM  sites  are  also  determined 
randomly  for  each  sortie.  Although  the  ranges  of  two  SAM  sites  are  different  from  each 
other,  the  search  patterns  and  the  probabilities  of  detection  and  kill  are  the  same. 
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Figure  5.  SAM  Positions  Submodel 


After  generating  air  defense  systems  and  strategic  targets,  the  Blue  side  attacking 
units  and  their  initial  positions  were  created  according  to  the  related  targets  (Figure  6). 
For  each  target,  four  Blue  A/G  attack  aircraft  are  created.  All  targets  are  placed  randomly 
in  the  range  of  the  air  defense  system.  This  implies  every  A/G  flight  has  to  enter  the  area 


Figure  6.  Targets  and  Strikers  Submodel 
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in  the  range  of  the  air  defense  system  and  become  vulnerable  to  missiles  of  air  defense 
systems.  At  the  same  time,  Red  air  defense  systems  have  to  be  exposed  to  Blue  HARM 
missiles  when  they  are  trying  to  defend  their  strategic  assets  by  operating  their  radars  and 
attacking  Blue  A/G  flights  with  their  missiles.  Thus  a  combat  environment  and  attrition 
for  both  sides  are  created  in  the  model. 

In  the  same  assignment  modules,  mathematical  calculations  are  made  for  attackers 
(Figure  7).  These  are  the  calculations  of  vector  velocities  of  attackers  that  help  to  move 
on  to  their  assigned  targets,  calculations  of  times  indicating  when  the  attackers  can  reach 
to  their  targets,  enter  the  threat  zone  and  exit  it.  The  A/G  aircraft  are  assumed  to  fly  at  a 
constant  velocity  of  480  knots  and  execute  a  low  altitude  operation  at  500  feet  above 
ground  level  (AGL).  A/G  flights  also  fly  in  an  offset  box  formation  to  make  a  time  and 
altitude  deconfliction  between  the  elements.  There  is  also  some  important  information 
gathered  for  SEAD  flights  to  help  them  generate  a  timeline  for  HARM  launches  in  the 
second  major  part  of  the  model. 


Figure  7.  Calculation  Assign  Modules  for  A/G  Aircraft 
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In  the  other  part  of  the  model,  the  entities  are  Blue  HARMs  that  interact  with  the 
air  defense  systems  of  the  Red  side.  These  missiles  are  being  launched  from  SEAD 
aircraft  which  are  not  modeled  as  separate  entities.  SEAD  flight  carries  and  launches 
eight  missiles  at  each  run.  They  usually  do  not  enter  the  range  of  the  threat  and  get 
exposed  to  the  Red  air  defense  missiles.  They  are  assumed  to  form  an  imaginary  box  in 
the  air  which  is  called  a  SEAD  box  to  provide  deconfliction  with  the  other  Blue  flights. 
SEAD  aircraft  are  not  involved  with  the  battle  directly,  thus  they  are  not  vulnerable  to  the 
SAMs  of  the  Red  side. 

After  HARMs  are  created  as  entities  for  Blue  side,  there  is  a  decide  module  to 
demonstrate  the  probability  of  some  failures  with  HARM  missiles  or  SEAD  aircrafts. 

This  module  cancels  some  missiles  by  chance  and  shows  the  effect  of  an  unplanned 
failure  of  missiles  in  the  air  during  combat  (Figure  8). 
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Figure  8.  HARM  Failure  Rate  Submodel 
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Afterwards,  the  allocation  of  HARMs  to  every  SAM  site  is  accomplished  related 
to  the  A/G  vulnerable  times  calculated  in  the  first  part  of  the  model  (Figure  9).  Logic 
changes  are  associated  with  vulnerability  times  of  attackers  and  make  the  distribution  in 
three  different  ways.  The  eight  HARMs  are  divided  into  6  to  2,  5  to  3  and  4  to  4  missiles 
for  each  SAM  site.  This  process  also  contains  the  calculations  of  time  over  targets  (TOT) 
of  each  missile  against  Red  SAM  sites.  HARMs  are  not  launched  reactively.  The 
accuracy  and  flow  of  intelligence  information  and  electronic  order  of  battle  (EOB) 
updates  are  assumed  to  be  at  a  sufficient  degree  to  make  SEAD  flights  plan  their  shots 
prior  to  vulnerable  times  of  A/G  aircraft.  Thus,  all  eight  HARMs  are  already  launched 
even  if  both  SAM  sites  are  hit  by  previous  missiles. 


Figure  9.  HARM  Allocations  Submodel 
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SEAD  flight  takes  an  initial  launch  position  in  the  battle  area  according  to  the 
locations  of  the  Red  air  defense  units  for  the  first  HARM.  After  that  they  move  to  a  new 
position  in  a  calculated  time  which  is  related  to  the  velocity  of  SEAD  aircraft  and  the 
time  between  two  consecutive  HARM  shots  for  the  remaining  shots.  They  remain  in  their 
SEAD  box  while  they  are  making  their  orbits  and  preparing  for  new  launches  (Figure 
10). 


Figure  10.  SEAD  Box  Submodel 


SEAD  flight  determines  the  location  of  the  SEAD  box  to  bring  the  flight  as  near 
as  possible  to  both  SAM  sites  without  entering  Red  missile  ranges.  There  are  two  options 
to  determine  the  location  of  the  SEAD  box.  One  is  from  the  north  and  the  other  is  from 
the  east.  There  is  logic  to  determine  the  placement  of  the  box  according  to  the  locations 
and  ranges  of  SAM  sites  to  get  the  nearest  and  safe  position  (Figure  11). 
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Figure  11.  Logic  of  SEAD  Box  Location 

After  determining  all  the  first  locations  of  SAM  sites,  targets,  A/G  and  SEAD 
flights  and  getting  all  the  calculations  related  to  time  for  A/G  flight  and  HARM  TOTs, 
the  combat  begins.  The  search  and  detection  process  of  SAM  sites  continues  from  the 
first  aircraft’s  entering  time  to  threat  area  until  the  last  aircraft’s  egress  time  (Figure  12). 


Figure  12.  SAM  &  HARM  Search  Patterns  Submodel 
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In  this  scenario,  the  air  defense  units  are  attacked  by  only  SEAD  aircraft  carrying 
HARMs.  Other  types  of  military  units  playing  a  role  in  a  SEAD  mission  are  omitted. 
There  are  no  other  DEAD  or  A/G  attacking  assets  carrying  weapons  to  destroy  air 
defense  system  units.  EW  assets  only  protect  the  SEAD  flight  and  they  are  not  assigned 
to  destroy  enemy  air  defense  units. 

The  attackers  were  considered  with  their  conventional  or  modern  weapons  against 
only  Red  A/G  targets  in  the  range  of  air  defense  units.  SAM  sites  can  only  engage  the 
attacking  aircraft.  Red  air  defense  units  cannot  operate  all  the  time  because  of  the  threat 
of  HARMs.  A  stochastic  detection  model  was  developed  for  their  operation.  Their 
operational  time  is  simulated  by  a  triangular  distribution.  SAM  sites  are  assumed  to  get  a 
lock  on  only  one  target  and  launch  one  missile  at  a  time.  They  can  get  different  detection 
opportunities  referred  to  as  glimpses  related  to  time  intervals  between  search  patterns. 
There  is  also  another  associated  probability  that  varies  in  accordance  with  the  skill  level 
of  SAM  operators.  For  instance,  a  high  level  operator  needs  less  time  than  a  low  level 
operator  to  detect  and  get  a  lock  on  the  target. 

After  making  detection,  the  operators  launch  a  missile  to  hit  the  Blue  attackers. 
They  can  engage  just  one  target  at  each  shot.  The  probability  of  detection  depends  on  the 
distance  and  directions  of  the  aircraft  to  the  SAM  sites.  Once  entering  the  threat  area,  the 
Blue  attackers  have  a  high  probability  of  being  hit  by  SAMs  because  they  are  moving 
toward  the  threat.  The  aircraft  fly  on  a  smooth  surface  and  are  not  terrain  masked.  They 
are  not  assumed  to  make  any  defensive  maneuver  against  SAM  locks  and  launches  which 
results  in  more  attrition  for  the  Blue  side.  On  the  other  side,  while  executing  the  egress 
phase,  their  speed  will  be  higher  and  they  will  show  the  aft  of  their  aircraft  which 
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decreases  the  probability  of  detection.  Different  values  of  probability  of  detection  were 
applied  to  ingress  and  egress  phases  of  the  attack. 

3.5  Movement  and  Detection  in  DES 

Movement  and  detection  are  crucial  issues  that  should  be  taken  into  consideration 
while  building  a  combat  model  using  a  discrete  event  tool.  Most  of  the  time,  both 
movement  and  detection  have  been  done  in  time-stepped  models.  Time  does  not  advance 
in  regular  intervals  in  discrete  event  simulation  as  the  simulation  time  is  moved  to  the 
time  of  next  event.  Although  it  seems  hard  and  infeasible,  there  is  a  way  to  do  both 
movement  and  detection  in  a  discrete  event  approach  (Buss  and  Sanchez,  2005). 

In  this  research,  there  are  SEAD  flights  and  A/G  attackers  moving  after  the  start 
of  battle.  They  are  assumed  to  fly  at  a  constant  speed  of  480  knots  and  at  a  fixed  altitude 
of  20,000  feet  for  SEAD  aircraft  and  500  feet  for  attackers.  They  have  a  linear  two 
dimensional  motion  which  is  the  simplest  possible  movement  in  a  discrete  event 
simulation. 

Any  aircraft  starts  its  move  at  an  initial  position  xq  related  to  its  assigned  target  at 
the  beginning  time  of  the  battle  G  with  a  constant  velocity  vector  v.  The  velocity  vector  is 
computed  related  to  the  assigned  targets  of  the  aircraft  to  ensure  they  proceed  to  their 
targets.  Storing  initial  position,  time  to  start  moving,  and  velocity  vector  of  any  moving 
entity  are  enough  to  determine  the  new  location  of  the  aircraft.  The  new  location  of  any 
aircraft  at  time  t  will  be  computed  by  this  equation  of  motion: 

x0  +  {t-t())v  (1) 
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To  find  the  TOT  of  an  attacker,  relative  velocity  is  used.  Since  the  target  is  stationary  and 
its  velocity  is  zero,  the  equation  of  motion  relative  to  the  target  will  give  the  TOT: 

t  ~  %tgt  ^  ^2) 

There  is  no  need  to  store  current  locations  of  the  aircraft  at  all  times  since  they  are  not 
being  detected  every  second  of  the  simulation  time.  These  computations  are  made  only 
when  a  SAM  site  has  an  opportunity  to  detect  and  ask  the  location  of  the  aircraft. 

The  cookie-cutter  sensor  is  the  simplest  way  of  detection  in  a  discrete  event 
modeling,  and  is  used  in  this  study  (Buss  and  Sanchez,  2005).  Air  defense  units  should 
not  move  while  operating,  that  means  the  sensors  of  the  SAM  sites  are  stationary  and 
each  of  A/G  attacking  aircrafts  are  the  moving  targets.  Again  at  time  to  the  aircraft  starts 
at  point  x0  and  proceeds  with  velocity  vector  v  to  its  target.  It  is  important  to  note  that 
position  and  velocity  calculations  are  made  relative  to  the  sensor.  The  main  concern  is  to 
find  the  detection  time  G  at  which  the  aircraft  enters  the  sensor’s  range.  The  position  of 
the  aircraft  at  the  time  of  detection  is  given  by  the  following  formula. 

x0  +  tdv  (3) 

The  detection  will  occur  when  the  distance  between  aircraft  and  sensor  equals  the  range, 
R,  of  the  sensor.  Thus  equation  3  becomes 

|k+HI  (4) 


Then  by  completing  the  calculation  of  the  length  of  this  vector  gives  the  solution  to  tci 
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With  the  condition  that  the  expression  under  the  radical  being  non-negative,  this  equation 
will  give  two  real  and  positive  values  for  t.  The  smaller  value  is  the  answer  for  time 
detection  tci  and  the  bigger  one  is  for  the  egress  time  that  aircraft  exits  the  threat  range  te. 

There  is  only  one  exception  from  the  cookie-cutter  logic  in  this  model.  After  the 
aircraft  come  upon  their  targets  and  drop  their  weapons,  they  do  not  follow  the  same 
direction  in  the  threat  area  to  move  on  as  seen  in  the  Figure  13. 


Figure  13.  Cookie-Cutter  Detection 

To  decrease  the  vulnerability  and  the  chance  to  be  detected  by  SAM  sites,  they 
use  a  different  velocity  vector  named  egress  vector  in  the  model  to  accomplish  egress 
phase  as  soon  as  possible.  They  use  the  shortest  path  towards  the  safe  area  and  minimize 
the  total  unprotected  time  in  the  threat  zone. 

The  movement  of  A/G  flights  and  their  detection  by  SAM  sites  is  briefly 
described  here.  The  detection  of  SAM  sites  by  HARMs  follow  the  same  cookie-cutter 
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logic.  This  time  the  entity  in  the  sensor  role,  HARM,  is  moving,  and  the  player  in  the 
target  role,  SAM  sites,  are  stationary.  HARM  sensor  begins  to  search  the  location  of  the 
SAMs  at  a  calculated  amount  of  time  after  launch.  When  the  sensor  of  HARM  receives 
any  emission  by  the  threats,  it  begins  homing  to  the  target.  Once  the  homing  is  initiated, 
it  flies  a  dive  trajectory  and  anns  its  proximity  fuse  until  it  approaches  the  target  and  hits 
the  target.  HARM  also  has  flexible  logic  and  chooses  the  next  highest  priority  target  in  its 
target  list  if  it  doesn’t  detect  its  primary  target. 

3.6  Conclusion 

In  this  chapter,  we  defined  an  overview  of  a  simplified  scenario  of  a  SEAD 
mission.  We  described  the  reasons  for  selecting  the  software,  steps  taken  while  building 
our  model  in  a  DES  environment,  a  brief  detail  of  the  model  structure,  and  the 
assumptions  made  to  make  the  model  reasonable.  Results  and  analysis  from  our  model 
are  discussed  in  the  next  chapter. 
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IV.  Results  and  Analysis 


4.1  Introduction 

The  previous  chapter  defined  the  model  built  for  this  research.  This  chapter 
includes  model  results  and  analysis.  In  the  first  section,  the  factors  and  output  data  used 
in  the  model  are  introduced  and  determining  the  appropriate  length  and  number  of 
replications  of  the  model  to  produce  these  output  data  is  explained.  The  following 
sections  include  the  design  of  experiments  (DOE)  and  regression  analysis  of  the  main 
model,  comparisons  made  between  different  competing  systems  on  the  basis  of  key 
performance  measures,  and  the  analysis  of  responses  from  the  model. 

4.2  Measures  of  Effectiveness 

Various  numbers  of  outputs  could  be  obtained  from  a  mission  level  simulation. 
The  detailed  model  can  easily  give  different  perfonnance  measures  from  the  results.  In  a 
combat  environment,  every  unit  has  distinctive  key  measures  to  calculate  their  own 
performances.  The  performance  values  of  each  mission,  number  of  attritions,  number  of 
ammunitions  fired,  number  of  targets  detected,  and  number  of  targets  destroyed  are  the 
most  usual  measures  of  effectiveness  (MOE)  in  combat  modeling.  In  this  study,  some  of 
the  outputs  we  captured  are  mission  success  for  each  mission,  survivability  score,  overall 
success  which  is  a  combination  of  mission  success  values  and  survivability  score,  killed 
SAMs,  detected  A/G  aircraft,  killed  A/G  aircraft,  killed  targets,  number  of  HARMs  fired, 
total  vulnerability  time  for  A/G  aircraft,  total  coverage  time  provided  by  SEAD  flight. 
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Overall  success  (OS)  is  one  of  the  most  important  MOEs  considered  in  this 
research.  As  mentioned  before,  overall  success  is  a  calculated  combination  of  the  scores 
of  each  mission  success  and  survivability.  The  mission  success  is  calculated  according  to 
the  number  of  targets  killed  by  A/G  strikers.  The  survivability  score  is  another  measure 
which  gives  the  number  of  A/G  strikers  alive  at  the  end  of  each  sortie.  Although  the 
military  commanders  usually  determine  the  weights  of  these  measures  related  to  the 
importance  level  of  each  one  in  the  combat,  the  largest  weight  (75%)  is  given  to  mission 
success.  These  measures  could  practically  be  changed  in  the  code  when  it’s  required. 

The  objective  for  half  width  variation  for  the  mean  of  key  MOEs  is  plus  or  minus 
1%.  The  main  model  was  run  for  10  days  and  three  sorties  were  accomplished  for  each 
day  which  gives  us  30  sorties  per  one  replication.  The  length  of  one  replication  was 
determined  related  to  the  duration  of  operational  exercises  such  as  Red  Flag  or  Anatolian 
Eagle.  First  we  ran  the  model  for  ten  replications  and  captured  an  estimated  variance  to 
implement  that  value  into  the  following  equation.  This  formula  assumes  that  as  we 
increase  the  number  of  replications  our  estimate  of  the  population  variance  will  not 
change  and  we  can  reach  an  approximate  expression  for  the  total  number  of  replications 
required  to  achieve  a  desired  half-width. 

nJ(P)=  min ji  >  n  :  <  /?  J  (6) 

na  is  the  number  of  additional  replications  needed  to  obtain  a  half  width  which  is  less 
than  or  equal  to  [i  which  is  equal  to  one  in  this  study.  S2 (n)  denotes  the  variance  with  the 
present  replication  number  and  i  denotes  the  iterative  increase  in  the  number  of 
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replications.  The  number  of  replications  were  iteratively  increased  and  finally  reached  the 
value  of  25. 


4.3  Design  of  Experiments  and  Regression  Analysis 

After  determining  the  number  of  replications  and  the  replication  length,  the  level 
of  critical  factors  was  detennined  that  allow  examination  of  the  varying  outputs.  Four 
different  main  factors  affected  the  outputs  directly  in  the  model.  These  factors  are  SAM 
on-air  rate  (SOR),  HARM  failure  rate  (HFR),  skill  level  of  SAM  operators  (SLO)  in 
terms  of  seconds  to  react  to  A/G  aircraft,  and  the  number  of  A/G  targets  for  each  SAM 
site  (NTG).  These  factors  have  two  different  levels  for  their  low  and  high  values.  A  2k 
factorial  design  is  constructed  to  detennine  which  factor  has  the  greatest  impact  on  the 
process  and  the  key  MOE  OS.  Arena®’s  Process  Analyzer  is  used  to  capture  the 
necessary  outputs.  The  main  factors  and  their  low  and  high  values  are  shown  in  Table  1. 
Only  NTG  used  a  center  level  with  a  value  of  4.  A  3  x2  factorial  design  was  constructed 
(Table  2). 


Table  1.  Main  Factors  and  Levels 


Factors 

Low 

High 

SOR 

50% 

100% 

HFR 

1% 

25% 

SLO 

30  sec. 

10  sec. 

NTG 

2 

6 
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Table  2.  Design  Points 


NTG 

SOR 

SLO 

HFR 

2 

50% 

10  sec. 

1% 

2 

50% 

10  sec. 

25% 

2 

50% 

30  sec. 

1% 

2 

50% 

30  sec. 

25% 

2 

100% 

10  sec. 

1% 

2 

100% 

10  sec. 

25% 

2 

100% 

30  sec. 

1% 

2 

100% 

30  sec. 

25% 

4 

50% 

10  sec. 

1% 

4 

50% 

10  sec. 

25% 

4 

50% 

30  sec. 

1% 

4 

50% 

30  sec. 

25% 

4 

100% 

10  sec. 

1% 

4 

100% 

10  sec. 

25% 

4 

100% 

30  sec. 

1% 

4 

100% 

30  sec. 

25% 

6 

50% 

10  sec. 

1% 

6 

50% 

10  sec. 

25% 

6 

50% 

30  sec. 

1% 

6 

50% 

30  sec. 

25% 

6 

100% 

10  sec. 

1% 

6 

100% 

10  sec. 

25% 

6 

100% 

30  sec. 

1% 

6 

100% 

30  sec. 

25% 

After  25  replications  of  the  model  for  each  24  design  points,  key  response  variable 
OS  values  were  collected.  All  these  input  and  output  variables  are  implemented  in  a 
multiple  linear  regression  model  to  find  out  the  relationship  between  these  variables  and 
the  response  variable.  Multiple  linear  regression  model  attempts  to  find  out  this 
relationship  by  fitting  a  linear  equation  to  observed  data.  This  linear  equation  provides  a 
regression  line  which  describes  how  the  mean  response  changes  with  explanatory 
variables.  The  observed  values  for  response  variable  vary  about  their  means  and  are 
assumed  to  have  the  same  standard  deviation.  The  fitted  values  estimate  the  parameters  of 
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the  population  regression  line.  Since  the  observed  values  vary  about  their  means,  the 
multiple  regression  model  includes  residuals  for  this  variation.  The  residuals  are  the 
deviations  of  the  observed  values  from  their  means,  which  are  assumed  to  be  normally 
and  independently  distributed  with  a  mean  of  zero  and  some  constant  variance.  These 
assumptions  are  checked  later. 

By  using  Minitab  statistical  package,  the  stepwise  regression  technique  is  applied 
to  determine  which  variables  have  a  significant  contribution  to  the  multiple  regression 
linear  model.  Four  main  factors  in  the  model  were  applied  first  and  had  a  very  low 
predictive  model.  Interactions  were  added  between  these  variables  to  obtain  a  more 
predictive  model.  Adding  new  variables  to  a  regression  equation  will  always  increase  our 
R  value,  which  gives  the  proportion  of  the  variability  in  the  response  that  is  fitted  by  the 
model,  even  when  the  new  variables  have  no  predictive  capability.  However,  the  adjusted 
R“  value  corrects  this  difficulty.  When  new  variables  are  added  to  the  regression 
equation,  the  adjusted  R“  value  does  not  increase,  if  the  new  variables  have  no  additional 
predictive  capability. 

Other  useful  exploratory  analysis  tools  for  factorial  experiments  include  main 
effects  plots  and  interaction  plots.  Figure  14  shows  the  main  effects  plot  for  the  response. 
This  plot  provides  the  information  about  how  a  factor  contributes  to  the  model  without 
any  interaction  between  the  other  factors. 

The  end  points  of  the  lines  are  the  mean  of  response  values  at  high  or  low  levels 
of  that  factor’s  design  points.  The  change  in  the  mean  of  responses  between  levels  of  a 
factor  is  illustrated  through  the  slopes  of  the  lines.  Usually  slopes  of  the  lines  give  the 
main  idea  of  significance  level  of  a  factor  on  the  response  in  these  kinds  of  plots.  A  steep 
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increase  between  the  means  indicates  that  a  factor  has  a  significant  effect  on  the  response 
variable.  A  gradual  change  ends  up  with  the  conclusion  of  a  minor  effect  on  the  response. 
HFR  is  as  a  good  example  of  a  gradual  slope.  HARM  failure  increases  make  little  change 
in  overall  success.  On  the  contrary,  the  other  three  factors  show  significant  effects  on  the 
response  with  their  steeper  slopes.  For  NTG,  increase  in  the  number  of  targets  gives  a 


better  result  in  the  response.  Although  this  increase  generates  new  increases  in  the 
number  of  A/G  aircraft  and  in  the  risk  of  attrition  rate,  it  concludes  a  better  score  in 
overall  success. 


interesting  behavior.  When  the  duration  of  on-air  time  of  a  SAM  site  increases,  it  will 
make  a  decrease  in  the  response.  Although  the  vulnerability  of  SAMs  against  HARMs 
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goes  up,  the  response  (overall  success)  drops  off.  Obviously  SLO  has  the  greatest  effect 
on  the  response.  As  expected,  the  higher  skill  level  makes  a  greater  decrease  in  the 
response.  On  the  contrary,  lower  skill  level  is  almost  completely  unsuccessful. 

Figure  15  shows  the  interactions  between  two  factors  among  each  other  and  the 
response.  The  different  shaded  lines  stand  for  each  level  of  the  first  factor  among  two 
factors  examined.  The  end  points  of  each  line  represent  the  two  levels  of  the  second 
factor.  The  values  at  those  end  points  correspond  to  the  response  values  depending  on 
these  two  factors. 

In  the  first  subplot,  we  examined  the  NTG  and  SOR  factors.  There  are  three  levels 
with  three  lines  for  NTG.  The  low  and  high  levels  of  SOR  are  the  end  points  of  those 
lines.  Their  interaction  between  each  factor  detennines  the  slope  of  that  line  and 
corresponding  values  on  the  right  hand  side  represent  the  response  variable  OS.  When 


I  nter action  Plot  for  OS 

Data  Means 


50  100  10  30  1  25 


HFR 


Figure  15.  Interaction  Plot  for  OS 
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NTG  is  at  its  low  level  which  is  the  black  line  in  this  case,  the  high  level  of  SOR  is  highly 
effective  and  make  a  great  decrease  in  the  response.  When  NTG  level  is  increased,  the 
same  decrease  in  the  response  is  observed  as  when  higher  levels  of  SOR  are  observed, 
but  not  as  sharp  slopes  as  seen  in  the  first  one.  When  SAM  sites  turn  their  systems  on 
during  the  whole  combat  they  can  be  more  successful  against  the  strikers,  but  their 
success  is  more  evident  with  a  low  numbers  of  targets.  When  the  number  of  targets 
increases,  SAM  sites  cannot  find  enough  time  and  chance  to  engage  each  target. 

In  the  second  subplot,  we  examined  NTG  and  SLO  factors  interaction  on  the 
response.  The  value  of  10  stands  for  the  high-skilled  SAM  operators  and  30  for  low- 
skilled  ones.  High  level  operators  have  a  great  impact  on  the  decrease  of  overall  success. 
As  expected,  low  level  SLO  can’t  be  as  successful  as  the  high  level.  The  same  result  as 
seen  in  the  previous  subplot  provides  the  same  conclusion.  When  the  number  of  targets 
increases,  the  success  for  SAM  sites  will  decrease.  This  provides  the  result  of  defending 
more  than  one  target  with  one  SAM  site  makes  operators  too  busy  to  engage  every  target. 

In  the  third  subplot,  NTG  and  HFR  factors  are  analyzed.  There  is  almost  no  slope 
for  each  line  which  means  the  HARM  failure  rate  does  not  have  a  significant  effect  on  the 
response.  The  only  change  among  the  three  lines  comes  naturally  from  the  NTG  factor,  as 
discussed  the  general  impact  of  the  change  in  the  number  of  targets  before.  The  same 
ineffectiveness  can  be  observed  from  the  other  subplots  of  HFR  on  the  third  column. 
Independent  of  the  other  factors,  HFR  cannot  make  big  differences  in  the  response.  In 
this  case,  being  limited  to  only  eight  HARMs  on  each  sortie  and  the  failure  rate  values 
between  the  values  of  one  and  25%  do  not  make  a  crucial  impact  on  the  survivability  of 
strikers. 
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The  last  interaction  plot  examined  is  the  combination  of  SOR  and  SLO  factors. 
The  higher  level  in  SLO  will  give  the  worse  result  in  response  again.  Also  the  high  level 
in  SOR  will  make  the  same  impact  on  the  response  by  decreasing  the  percentage  of 
overall  success.  This  means  despite  the  fact  that  being  more  vulnerable  to  HARMs,  SAM 
sites  could  find  more  chances  to  detect  and  kill  their  targets. 

The  interaction  between  variables  was  also  analyzed.  Table  3  shows  the 
correlation  values. 


Table  3.  Correlation  between  the  Variables 


NTG 

SOR 

SLO 

HFR 

SOR*SLO 

SOR*HFR 

SOR*NTG 

NTG*SLO 

NTG*HFR 

SLO*HFR 

NTG 

1.00 

SOR 

0.00 

1.00 

SLO 

0.00 

0.00 

1.00 

HFR 

0.00 

0.00 

0.00 

1.00 

SOR*SLO 

0.00 

0.53 

0.80 

0.00 

1.00 

SOR*HFR 

0.00 

0.32 

0.00 

0.90 

0.17 

1.00 

SOR*NTG 

0.75 

0.61 

0.00 

0.00 

0.33 

0.20 

1.00 

NTG*SLO 

0.60 

0.00 

0.74 

0.00 

0.59 

0.00 

0.45 

1.00 

NTG*HFR 

0.38 

0.00 

0.00 

0.86 

0.00 

0.77 

0.28 

0.23 

1.00 

SLO*HFR 

0.00 

0.00 

0.44 

0.80 

0.35 

0.72 

0.00 

0.32 

0.69 

1.00 

The  correlation  between  “SOR*HFR”  and  “HFR”  is  0.90.  Since  they  are  highly 
correlated,  addition  of  the  variable  “SOR*HFR”  may  not  significantly  improve  the 
model.  The  other  variables  were  also  examined  in  the  same  way.  After  fitting  the 
regression  line  to  this  equation,  it  is  important  to  investigate  the  residuals  which  are  the 
differences  between  the  observed  and  predicted  values  to  determine  whether  or  not  they 
appear  to  fit  the  assumption  of  a  normal  distribution.  Normality  is  one  of  the  three  basic 
assumptions  of  these  residuals.  A  normal  probability  plot  of  the  standardized  residuals  is 
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shown  in  the  Figure  16.  Despite  two  small  light  departures  on  both  tails  in  the  data,  the 
residuals  do  not  seem  to  deviate  from  a  normal  distribution  in  any  systematic  manner. 


-1 - 1 - 1 - 1 - !- 

-2-10  1  2 

Standardized  Residual 


Figure  16.  Normal  Probability  Plot  of  Residuals 
Residuals  can  be  thought  of  as  elements  of  variation  unexplained  by  the  fitted 
model.  Thus  the  other  basic  assumption  about  residuals  is  constant  variance  is  checked  by 
a  scatter  plot,  the  residuals  against  the  fitted  values  (Figure  17). 


Versus  Rts 


Figure  17.  Residual  plots  vs.  Fitted  Values 
Plotting  residuals  versus  the  value  of  a  fitted  response  should  produce  a 
distribution  of  points  scattered  randomly  about  zero,  regardless  of  the  size  of  the  fitted 
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value.  If  residual  values  increase  as  the  size  of  the  fitted  value  increases,  the  residual 
cloud  becomes  "funnel  shaped"  with  the  larger  end  toward  larger  fitted  values  which 
means  the  residuals  have  a  non-constant  variance.  Although  the  residuals  make  a  dense 
distribution  between  the  values  60  and  100,  a  funnel  shaped  residual  cloud  is  not 
observed.  The  scatter  in  the  residuals  between  60  and  80  is  similar  to  the  scatter  in  the 
residuals  between  80  and  100.  This  suggests  that  the  standard  deviation  of  the  residuals  is 
roughly  constant  for  the  responses  observed  at  each  value. 

After  these  steps,  the  regression  model  is  reached.  Minitab  also  provides  a 
parameter  table  shown  in  Table  4  which  helps  to  understand  the  variables  that  make  a 
contribution  to  the  model  at  different  levels. 

OS  =  40.08  +  2.33 NTG  -  0.23 SOR  +  2.15SLO  +  0A6HFR  -  0.01SGR  *  SLO  + 

0.06 SOR  *  NTG  -0.1 5 NTG  *  SLO  (7) 


Table  4.  Parameter  Estimates 


Predictor 

Coef.  (p) 

Std.Err.of  Coef. 

p-values 

Intercept 

40.082 

4.143 

0.000 

NTG 

2.3257 

0.7864 

0.010 

SOR 

-0.22849 

0.04790 

0.000 

SLO 

2.7469 

0.1373 

0.000 

HFR 

0.15828 

0.09044 

0.101 

SOR*SLO 

-0.011166 

0.001373 

0.000 

SOR*NTG 

0.056522 

0.008407 

0.000 

NTG*SLO 

-0.14742 

0.02102 

0.000 

As  seen  on  the  first  column,  along  with  four  main  factors,  three  interaction 
variables  also  help  to  predict  the  response.  The  regression  coefficients  are  shown  in  the 
second  column.  The  third  column  contains  the  standard  errors  of  the  regression 
coefficients  which  can  be  used  for  hypothesis  testing  and  constructing  confidence 
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intervals.  P-values  or  the  significance  levels  for  t  statistics  in  the  last  column  tell  whether 
a  variable  has  statistically  significant  predictive  capability  in  the  presence  of  the  other 
variables.  A  p-value  smaller  than  0.05  means  that  variable  is  statistically  significant  in  the 
model  at  the  a  =  .05  level.  In  some  circumstances,  a  non-significant  p-value  might  be 
used  to  detennine  whether  to  remove  a  variable  from  a  model  without  significantly 
reducing  the  model's  predictive  capability.  HFR  has  a  non-significant  p-value,  however 
when  it  is  removed,  the  model  is  less  significant  and  the  normality  plot  has  larger 
deviations  than  the  present  one.  These  p-values  should  not  be  used  to  eliminate  more  than 
one  variable  at  a  time.  A  variable  that  does  not  have  predictive  capability  in  the  presence 
of  the  other  predictors  may  have  predictive  capability  when  some  of  those  predictors  are 
removed  from  the  model. 

Analysis  of  Variance  (ANOVA)  table  (Table  5)  explains  the  variability  in  the 
response  variable.  The  amount  of  variability  can  be  measured  by  the  Total  Sum  of 
Squares.  The  ANOVA  table  partitions  this  variability  into  two  parts.  One  portion  is  fitted 
by  the  regression  model  and  labeled  as  Regression  Sum  of  Squares.  It’s  the  reduction  in 
uncertainty  that  occurs  when  the  regression  model  is  used  to  predict  the  responses.  The 
remaining  portion  is  the  uncertainty  that  remains  even  after  the  model  is  used  and  labeled 
as  Residual  Error  Sum  of  Squares.  The  model  is  considered  to  be  statistically  significant 
if  it  can  account  for  a  large  amount  of  variability  in  the  response. 


Table  5.  Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

p-value 

Regression 

8 

6569.45 

821.18 

290.50 

0.000 

Residual  Error 

15 

42.40 

2.83 

Total 

23 

6611.85 
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Mean  Squares  are  the  Sums  of  Squares  divided  by  the  corresponding  degrees  of 
freedom.  The  F  Ratio  is  the  test  statistic  used  to  decide  whether  the  model  as  a  whole  has 


statistically  significant  predictive  capability.  The  null  hypothesis  states  that  all  regression 
coefficients  are  equal  to  zero.  In  other  words  the  model  has  no  predictive  capability.  The 
large  values  of  F  statistic  provide  evidence  against  null  hypothesis  and  at  least  one  of  the 
coefficients  is  different  from  zero.  The  p-value  for  the  F  statistic  is  less  than  0.001, 
providing  strong  evidence  against  the  null  hypothesis. 

R  value  is  the  squared  multiple  correlation  coefficient  and  gives  the  proportion  of 
the  variability  in  the  response  that  is  fitted  by  the  model.  In  this  regard,  if  a  model  has  a 
perfect  predictability  R  is  equal  to  1.  The  Summary  of  Fit  shows  these  values  in  Table  6. 
As  mentioned  before  the  adjusted  R  value  makes  a  correction  to  the  increase  in  R“  value 
when  new  variables  are  added  that  have  no  additional  predictive  capability  to  the  model. 
In  this  case,  99%  of  the  variance  in  the  response  variable  (OS)  is  explained  by  the  model. 
The  Root  Mean  Square  Error  is  the  square  root  of  the  Residual  Mean  Square.  It  is  the 
standard  deviation  of  the  data  about  the  regression  line,  rather  than  about  the  sample 
mean. 


Table  6.  Summary  of  Fit 


R  Square 

0.994 

R  Square  adjusted 

0.990 

Root  Mean  Square  Error 

1.683 

4.4  Comparison  of  Different  Systems 

In  this  section,  statistical  analyses  of  the  output  from  two  different  versions  of  the 
main  model  that  might  represent  competing  system  designs  is  discussed.”  The  real  utility 
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of  simulation  lies  in  comparing  output  of  these  alternative  systems”  (Law,  2007:548).  In 
this  sense,  appropriate  statistical  methods  are  essential  in  making  correct  conclusions. 
Two  options  to  construct  confidence  intervals  for  the  difference  between  two 
performance  measures  are  available.  One  is  the  two-sample-t  approach  which  requires 
independence  and  equal  variances  but  not  equal  sample  size  between  two  systems. 
However,  equality  of  variances  might  not  necessarily  be  a  good  assumption  when 
simulating  real  world  systems  such  as  mission  level  combat  systems.  Thus  a  paired-t  test 
is  the  other  option.  The  advantage  of  this  approach  is  it  does  not  require  equal  variance 
and  independence  between  systems.  The  sample  sizes  should  be  equal  in  this  approach. 
Another  consideration  is  using  common  random  numbers  (CRN)  to  achieve  significant 
variance  reduction  (Law,  2007:555).  The  same  random  number  streams  and  seeds  is  used 
for  each  system  to  synchronize  our  use  of  random  numbers.  Since  this  approach 
intentionally  creates  dependence  between  the  systems  as  a  variance  reduction  technique, 
it  requires  paired-t  test  approach  to  construct  confidence  intervals  for  the  difference 
between  two  perfonnance  measures. 

Two  different  systems  are  used  in  this  study.  In  the  first  one,  all  A/G  aircraft  use 
the  same  exact  time  to  be  over  their  targets.  However,  it  provides  different  times  for 
strikers  to  cross  the  forward  edge  of  the  battle  area  (FEB  A).  In  the  other  one,  all  strikers 
use  same  exact  time  to  cross  FEB  A  and  this  naturally  provides  different  TOTs  for  A/G 
aircraft.  The  first  system  is  named  as  System  A,  and  the  second  one  as  System  B. 

These  systems  are  examined  based  on  how  these  two  different  fonnations  effect  the 
vulnerability  time  of  strikers  in  the  range  of  SAM  sites  and  the  dispersion  of  HARM 
TOTs  which  are  vital  for  strikers’  survivability  and  our  response  variable  OS. 
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First,  how  different  TOTs  and  attack  methods  of  these  systems  affect  vulnerability 
time  was  examined.  The  same  design  points  for  both  systems  were  chosen  and  a  paired-t 
test  between  the  means  was  performed. 

In  Table  7,  the  first  line  defines  the  design  points.  The  letters  stand  for  the  system 
name,  the  numbers  represent  NTG,  SOR,  SLO,  HFR  factors  in  order.  First  the  value  of 
two  for  NTG  was  examined  which  means  there  are  only  two  targets  in  the  range.  The 
other  factors  don’t  have  any  effect  on  vulnerability  times.  As  seen  on  the  first  two 


Table  7.  Means  of  Vulnerability  Times  For  Two  Design  Points  of  Each  System 


A-2-50-10-1 

B-2-50-10-1 

A-6-50-10-1 

B-6-50-10-1 

396.744 

396.758 

521.061 

493.385 

382.531 

382.546 

503.108 

476.828 

357.737 

357.753 

475.343 

461.608 

369.602 

369.615 

500.428 

477.512 

403.936 

403.950 

503.038 

470.362 

356.818 

356.832 

482.712 

459.474 

346.143 

346.156 

477.994 

451.544 

359.698 

359.714 

484.514 

459.407 

355.521 

355.538 

496.863 

467.177 

355.167 

355.181 

476.597 

456.672 

387.925 

387.938 

501.054 

474.771 

347.650 

347.665 

489.361 

464.104 

389.119 

389.135 

514.210 

478.494 

366.425 

366.439 

482.421 

460.833 

404.301 

404.314 

510.993 

486.652 

369.537 

369.550 

474.874 

448.083 

348.729 

348.744 

488.526 

461.234 

371.098 

371.113 

483.558 

459.106 

388.203 

388.214 

505.566 

477.638 

345.134 

345.149 

470.040 

453.691 

401.593 

401.606 

479.007 

462.224 

371.727 

371.742 

490.940 

477.207 

325.201 

325.215 

455.724 

432.564 

399.275 

399.291 

505.694 

474.657 

370.464 

370.478 

506.438 

477.860 
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columns,  the  mean  values  are  very  close  to  each  other.  But  the  results  of  paired-t  test 
(Table  8)  which  is  equivalent  to  testing  the  null  hypothesis  H0:  pA  -  Pb  =  0,  indicates 
rejection  of  the  null  hypothesis  at  the  a  =  .05  level.  The  confidence  interval  does  not 
contain  zero  and  the  p-value  is  smaller  than  a  value.  Although  vulnerability  times  of 
systems  are  statistically  different  from  each  other,  the  difference  at  the  second  decimal 
place  in  the  vulnerability  times  relating  to  the  systems  is  clearly  not  practically  significant 
when  there  are  only  two  targets  in  the  range. 

Table  8.  Paired-t  Test  for  Vulnerability  Times  (A-2-50-10-1,  B-2-50-10-1) 


N 

Mean 

StDev 

SE  Mean 

A-2-50-10-1 

25 

370.81 

21.49 

4.30 

B-2-50-10-1 

25 

370.83 

21.49 

4.30 

Difference 

25 

-0.014270 

0.001394 

0.000279 

95%  Cl 

(-0.014845,-0.013695) 

P- Value 

0.000 

In  the  third  and  fourth  column  of  Table  7,  the  mean  values  of  vulnerability  times 
for  both  systems  at  the  value  of  six  for  NTG  are  shown  along  with  results  of  the  paired-t 
test  (Table  9). 

Table  9.  Paired-t  Test  for  Vulnerability  Times  (A-6-50-10-1,  B-6-50-10-1) 


N 

Mean 

StDev 

SE  Mean 

A-6-50-10-1 

25 

491.20 

15.79 

3.16 

B-6-50-10-1 

25 

466.52 

13.32 

2.66 

Difference 

25 

24.68 

5.47 

1.09 

95%  Cl 

(22.42,  26.94) 

P- Value 

0.000 

The  null  hypothesis  that  these  two  vulnerability  times  are  statistically  same  is 
rejected.  But  this  time  the  confidence  interval  lies  well  above  zero  and  is  larger  than  the 
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first  one.  This  result  indicates  that  A/G  aircraft  in  System  A  spend  more  time  in  the 
vulnerability  area  than  A/G  aircraft  in  System  B.  First  it  could  be  thought  that  all  strikers 
in  System  A  go  into  and  out  of  SAM  area  at  once  and  they  have  to  spend  less  time  in  the 
target  area.  Since  each  A/G  target  location  has  a  different  distance  to  SAM  site,  and  the 
vulnerability  time  window  is  calculated  from  the  very  first  striker’s  entering  time  to  the 
threat  zone  until  the  very  last  striker’s  exit  time,  it  provides  a  dispersed  and  larger 
exposed  time  window  for  strikers. 

Since  the  numbers  of  SAM  sites  and  A/G  targets  are  not  big  values  such  as  two 
for  SAMs  and  three  for  A/G  targets  for  each  SAM,  and  the  maximum  range  of  SAM  sites 
is  limited  to  25,  the  difference  between  two  means  cannot  be  thought  practically 
significant.  But  if  these  values  are  increased,  the  difference  between  means  will  get 
higher  and  begin  to  make  things  more  difficult  for  attackers. 

After  finishing  systems’  effect  on  vulnerability  time,  the  difference  between  the 
systems  influences  our  key  MOE  overall  success  is  discussed.  Sixteen  design  points  for 
each  system  are  chosen.  These  points  include  the  low  and  high  values  of  four  factors.  By 
applying  paired-t  tests  to  these  points,  the  following  results  were  observed.  Means  and 
standard  deviations  of  each  system,  95%  confidence  intervals  and  p-values  are  listed  in 
Table  10. 

In  the  first  two  lines,  the  systems  indicate  that  they  are  statistically  different  from 
each  other  since  95%  CIs  don’t  cover  zero  and  the  p-values  are  smaller  than  a.  For  both 
design  points  System  B  shows  better  performance  than  System  A.  The  only  difference 
between  the  two  design  points  is  HFR.  HFR  does  not  have  a  significant  effect  on  OS.  The 
ineffectiveness  of  HFR  is  discussed  later. 
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Table  10.  Paired-t  Test  for  OS 


Design  Points 

System  A 

System  B 

no 

NTG-SOR-SLO-HFR 

mean 

std  dev 

mean 

std  dev 

95%  Cl 

p-value 

1 

6-50-10-1 

72.776 

2.556 

76.918 

1.981 

-5.001  ,-3.281 

0.000 

2 

6-50-10-25 

73.176 

2.385 

74.939 

2.346 

-2.725  ,  -0.802 

0.000 

3 

6-50-30-1 

98.000 

0.531 

98.188 

0.507 

-0.468,  0.092 

0.177 

4 

6-50-30-25 

97.898 

0.544 

97.748 

0.581 

-0.220,  0.521 

0.409 

5 

6-100-10-1 

70.700 

2.089 

70.935 

2.574 

-1.546,  1.076 

0.714 

6 

6-100-10-25 

66.521 

3.122 

69.554 

2.791 

-4.811  ,-1.257 

0.002 

7 

6-100-30-1 

86.531 

1.004 

87.764 

1.142 

-1.672,-0.795 

0.000 

8 

6-100-30-25 

84.225 

1.173 

86.178 

1.186 

-2.506,-1.400 

0.000 

9 

2-50-10-1 

56.320 

5.613 

62.080 

4.522 

-7.688  ,  -3.832 

0.000 

10 

2-50-10-25 

57.218 

5.111 

60.963 

4.784 

-5.940,-1.550 

0.002 

11 

2-50-30-1 

96.802 

1.068 

96.213 

1.308 

-0.108,  1.285 

0.094 

12 

2-50-30-25 

96.255 

1.318 

96.027 

1.296 

-0.557,  1.014 

0.554 

13 

2-100-10-1 

45.637 

3.354 

51.313 

5.499 

-8.260,-3.100 

0.000 

14 

2-100-10-25 

42.547 

2.653 

49.113 

4.720 

-8.483  ,  -4.650 

0.000 

15 

2-100-30-1 

72.063 

2.562 

71.990 

3.878 

-1.653,  1.800 

0.930 

16 

2-100-30-25 

67.257 

3.900 

68.935 

3.779 

-3.218,-0.139 

0.034 

In  the  following  two  lines,  a  decrease  in  SLO  from  the  first  two  lines  is  observed. 
A/G  aircraft  will  be  more  successful  when  the  skill  levels  of  operators  decrease.  The  rise 
in  the  mean  values  is  an  obvious  proof  of  this  development  and  these  OS  values  are  the 
highest  ones  among  all  these  design  points.  At  the  same  time,  it  cannot  be  said  this 
change  creates  a  difference  between  two  systems  as  seen  by.  Failing  to  reject  these 
systems  are  statistically  different  from  each  other  for  both  design  points  at  SLO  value  of 
30  sec. 

In  the  fifth  and  sixth  lines,  the  SOR  and  SLO  level  is  increased  to  their  high 
values.  In  line  five,  no  difference  between  two  systems  is  observed  but  in  line  six  the 
effect  of  HFR  on  OS  is  seen.  By  decreasing  OS  value  from  70.7  to  66.5,  HFR  provides  a 
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statistical  difference  between  System  A  and  System  B.  In  this  case,  HFR  acted  as  a  key 
factor  with  the  high  levels  of  SOR  and  SLO  on  changing  OS  value. 

In  lines  seven  and  eight,  no  statistical  difference  between  competing  systems  is 
observed.  The  major  reason  for  this  result  is  the  high  level  of  SOR  accompanying  with 
low  level  of  SLO.  As  mentioned  before,  the  high  level  of  SOR  make  an  interesting 
decrease  in  OS  although  SAM  sites  spend  more  time  exposed  to  the  threat  of  HARMs. 

The  remaining  lines  in  Table  9  are  a  repeat  of  the  same  factor  levels  in  the  top  of 
the  table  with  a  change  in  NTG  value.  The  number  of  targets  in  the  combat  area  is 
decreased  from  six  to  two.  In  lines  nine  and  ten,  no  statistical  difference  between  two 
systems  is  noted.  However,  comparing  the  mean  response  values  with  lines  one  and  two, 
there  is  a  great  decrease  in  OS  values.  It  indicates  that,  when  the  number  of  targets 
decreases  in  the  area,  SAM  operators  find  more  chances  to  engage  their  targets.  Thus, 
they  can  detect  and  kill  more  aircraft  which  results  in  lower  values  of  OS. 

In  lines  1 1  and  12,  the  highest  scores  of  OS  are  seen  again  after  six  target  versions 
of  these  design  points.  It  proves  the  same  idea  above  about  the  effect  of  the  number  of 
targets  on  OS.  The  same  results  for  the  rest  of  the  designs  are  observed.  Here  again,  the 
results  fail  to  reject  the  claim  that  these  systems  are  statistically  the  same.  In  lines  13  and 
14,  better  results  with  System  B  are  shown  and  in  lines  15  and  16,  a  significant  effect  of 
HFR  on  the  results  is  observed.  A  high  value  for  HFR  shows  System  B  is  better  in  the 
last  comparison. 

The  results  and  comments  are  discussed  next.  As  opposed  to  our  original 
expectations,  System  B  showed  better  performance  in  most  comparisons.  Although  the 
high  values  of  SOR  make  an  increase  in  exposed  time  to  HARMs,  the  overall  success 


56 


decreased.  When  the  number  of  A/G  targets  and  normally  the  number  of  A/G  aircraft 
decrease  in  the  range  of  SAM  sites,  SAMs  killed  more  and  gave  a  significant  damage  to 
Blue  side.  This  shows  the  importance  that  EW  assets  should  increase  suppression  with 
jamming  to  make  SAM  sites  busy  and  inoperative.  With  few  exceptions,  HFR  has  no 
effect  on  the  results.  Since  only  eight  HARMs  are  used  in  each  sortie  and  the  probability 
of  failure  rate  cannot  be  too  high,  this  result  was  no  surprise.  Finally  the  skill  level  of 
operators  could  be  vital  for  both  sides.  When  the  value  of  SLO  decreases,  Blue  side 
success  increases.  When  SLO  increases,  Red  inflicts  greater  damage  to  its  opponent. 

4.5  Conclusion 

In  this  chapter,  output  analysis  was  discussed.  First  the  appropriate  length  and 
number  of  replications  of  the  model  to  produce  necessary  outputs  was  determined.  The 
factors  contributing  to  the  model  were  then  examined  which  resulted  in  a  regression 
analysis  with  a  factorial  design.  Finally  the  results  of  different  competing  systems  were 
analyzed  and  conclusions  about  the  factors  in  the  model  were  drawn.  Chapter  5  will 
discuss  the  highlights  of  this  study. 
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V.  Conclusion 


5.1  Introduction 

The  previous  four  chapters  presented  the  research  that  was  undertaken.  First  a 
brief  summary  of  SEAD  mission  evolution  was  introduced  followed  by  a  literature 
review  on  simulation,  combat  modeling  and  previous  studies  on  related  subjects.  Next, 
important  model  details  and  information  was  provided.  The  last  section  analyzed  the 
outputs  of  the  model  and  provided  conclusions.  This  chapter  will  give  highlights  from  the 
previous  chapters  and  make  conclusions  and  recommendations  for  future  research. 

5.2  Summary  of  the  Research 

The  objective  of  this  research  was  to  build  a  responsive  and  flexible  model  using 
a  discrete  event  simulation  to  investigate  the  effectiveness  of  a  simplified  SEAD  scenario 
with  its  different  factors.  Thus  simulation  cannot  by  itself  be  a  perfect  representation  of 
real  world,  the  plan  was  to  build  a  mission  level  model  with  enough  details  to  draw 
conclusions.  First  the  important  factors  that  should  be  included  in  the  model  to  represent 
a  SEAD  mission  were  designed.  Several  assumptions  were  made  to  keep  the  model 
feasible  and  simple.  After  determining  the  entities  and  main  states  of  the  entities,  model 
construction  began. 

Movement,  detection  and  searching  were  the  major  challenges  for  a  combat 
model  in  an  event-stepped  simulation.  After  developing  the  main  model,  minor  changes 
were  made  to  the  model  to  mirror  real-world  situations.  These  models  involved  the 
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characteristics  of  our  design  points  which  were  used  in  regression  analysis.  The  models 
were  run  for  10  days  with  25  replications.  Outputs  from  these  different  models  were 
captured.  The  most  important  measure  of  effectiveness  which  is  OS  was  evaluated.  The 
study  concluded  with  analysis  and  comments  of  the  results. 

5.3  Conclusions  of  the  Research 

Four  main  factors  were  used  in  the  model.  These  factors  are  SAM  on-air  rate 
(SOR),  HARM  failure  rate  (HFR),  skill  level  of  SAM  operators  (SLO)  in  terms  of 
seconds  to  react  to  A/G  aircraft  and  the  number  of  A/G  targets  for  each  SAM  site  (NTG). 
The  key  MOE  was  determined  as  overall  success  (OS).  Also  two  different  systems  were 
built  to  make  comparisons.  In  System  A,  all  A/G  flights  use  only  one  TOT.  In  System  B, 
A/G  flights  use  different  TOTs  but  same  FEBA  crossing  time. 

The  results  of  the  research  show  that  when  the  exposure  times  of  Red  SAM  sites 
against  Blue  HARMs  increase,  the  higher  levels  of  SOR  always  decrease  Blue  OS  level. 
When  NTG  decreases,  the  success  of  Red  SAM  sites  proportionally  increases.  It  indicates 
that  when  the  busy  time  of  SAM  operators  and  systems  drops,  they  are  more  lethal 
against  Blue  forces.  Naturally  SLO  provides  results  as  expected  and  shows  that  skill  level 
of  operators  significantly  effects  system  performance.  Finally  HFR  is  the  most  ineffective 
factor  in  this  research.  Because  of  the  low  number  of  HARMs,  HFR  does  not  make  major 
effects  on  the  scores. 

Another  result  is  System  A  causes  Blue  A/G  aircraft  to  spend  more  time  in  the 
range  of  Red  SAM  sites  which  increases  the  vulnerability  time  of  System  A  over  System 
B.  Because  of  that  reason,  System  B  shows  a  higher  Blue  OS  in  most  of  the  runs. 
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5.4  Recommendations  for  Future  Study 

The  model  can  be  enhanced  by  increasing  the  scope  of  this  simulation.  A 
simplified  scenario  of  a  SEAD  mission  was  modeled.  The  number  of  SAM  sites,  targets, 
SEAD  and  A/G  flight  are  limited.  By  increasing  these  numbers  with  minor  logic  changes 
in  the  model,  more  representative  system  performance  could  be  captured.  Model  fidelity 
could  be  significantly  increased  by  adding  A/A,  EW  capabilities,  intelligence  and  EOB 
update  processes  in  support  of  these  missions. 
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